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Abstract of JP 2001119663 W 
PROBLEM TO BE SOLVED: To provide a 
multimedia information processing unit that 
automatically arranges Image information, depending 
on an acquired time and position and that exhlt)lts the 
arranged Information. SOLUTION: An Image 
acquisition section 1 acquires image Information of a 
still Image or a dynamic Image, a comment section 2 
enters a graphic, sound and a hand-written memo to 
be attached to the image information acquired by the 
Image acquisition section 1 as comments, a time 
counter section 3 coints the time when the image 
acquisition section 1 acquires the image information. 
Furthermore, a multimedia compatible storage 
section 4 stores the image information acquired by 
this image acquisition section 1 , the comments 
entered by the comnfient section 2, and the time 
"counted 6y"tTe"time counter sectlorTSIn cross-' 
reference with each other. Then an arrangement 
exhibit section 5 arranges the image information and 
the comments stored in the multimedia compatible 
storage section 4, depending on the time stored in 
cross-reference with the image Information and the 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the multimedia information processing 
unit and image information processing unit which gain and present the picture 
information of a still picture or an animation, and relates to the multimedia information 
processing unit and image processing device which can arrange picture information 
automatically by the time and the position which were acquired especially, and can be 
shown. 
[0002] 

[Description of the Prior ArtJWhat is called the digital still camera and digital movie cam 
that gain the picture information of a still picture or video have been variously developed 
by carrying out image formation of the object image on CCD with improvement in 
semiconductor manufacturing technology in recent years and an image processing 
technique. 

[0003]In the latest digital still camera, there are some which it not only can record a still 
picture, but can attach notes of a recording date, a sound, etc. to a still picture. On the 
other hand, a digital movie cam not only also records an animation and a sound, but has 

someJwhichxan-attachjM)tes.-of-aj;ecordingjdate,.a.title, etcAndJUSualLy these, digital still _ 

cameras and digital movie cams are equipped with the method of arranging so that it may 
be easy to manage the recorded still picture or animation, and notes. 
[0004]Here, it takes up as an example about the conventional arrangement method in a 
digital still camera and a digital movie cam with a notes option. This arrangement method 
can be classified as follows. 

[0005](1) A page turning-over type (2) list-display type (3) directory type (4) album type 
(1) page turning-over type, For example, it displays in order so that a page may be turned 
over from the still picture which recorded first the still picture currently recorded on the 
camera body by the digital still camera with a liquid crystal display, It is a method which 
reproduces notes of the soimd added to the still picture which pushed the sound button 
and was displayed, or has displayed enough the time which recorded the still picture 
which pushed tiie date button and was displayed on a liquid crystal display, and carries 
out it. 

[0006]In the case of a digital movie cam, an animation after pushing a record button until 
it stops record is divided as one animation, it displays it on a liquid crystal display by 
using the picture of the beginning of the divided animation as a representative picture 
image, and carries out page turning over of this representative picture image. Record and 
reproduction of a still picture or an animation are set up with a mode switching button, 
and the passing <a thing> on display of a still picture is performed by pushing a page 
turning-over button. 

[0007]In this page turning-over type, the still picture recorded in the camera body can be 
checked easily, and reproduction or a display of notes is also easy. However, since it was 
necessary to carry out page turning over repeatedly until the still picture currently looked 
for is found when a huge number of still pictures and animations are recorded, since page 
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turning over can be carried out for every sheet, there was a problem that search took time. 
[0008]The list display type of (2) transmits the still picture and notes which were 
recorded by the digital still camera to a personal computer, for example, It is the method 
of displaying notes of the time etc. which carried out the list display of the thumbnail 
image (reduction image), and recorded the transmitted still picture under each thumbnail 
image on the display of a personal computer. Reproduction of notes of a sound clicks and 
chooses with a mouse one of the thumbnail images by which the list display was carried 
out, and is performed by pushing a sound reproduction button. In the case of a digital 
movie cam, the thumbnail image of a representative picture image is used for a list 
display. 

[0009]In this list display type, the still picture or animation currently looked for can be 
easily specified from the tiiumbnail image by which the list display was carried out. 
However, when long term use of a digital camera with a notes option or the digital movie 
cam was carried out, for example, a huge number of still pictures and animations were 
recorded and it transmitted to a personal computer, the still picture or animation currently 
looked for out of a huge number of list displays had to be found, and the user's burden 
was heavy. 

[0010]There is JP,10-91585,A (a portable information processor and an information 
processing method for the same) as another list display type example, for example. This 
device incorporates schedule contents as a still picture with a camera, While displaying 
and carrying out the list display of the mark which shows the existence of a schedule to 

^the-calendar-for one month-by-arranging-an icon (the- inside of a specification-schedule 

stamp) including information, including the time of a schedule, etc., on the still picture. 
The schedule contents containing the still picture incorporated by choosing this mark are 
displayed. 

It can be regarded as one of the methods which arranges the incorporated still picture. 

[001 IJThe still picture which expresses the contents of that schedule briefly can be 
searched with this device by carrying out the list display of the mark showing each 
schedule to the calendar of calendar state, and choosing one of marks of this as it. 
However, a still picture cannot be searched with this device from a schedule, and a still 
picture without a schedule cannot be searched vdth it. 

[00 12] When the directory type of (3) transmits a still picture and an animation to a 
personal computer with a list display type, it is transmitted to the directory which 
attached the name representing the still picture and animation, A list display type can be 
regarded as all the still pictures or animations being transmitted to one directory. After 
choosing whether the list display of the still picture or animation transmitted in which 
directory is carried out, the list display of it is carried out. 

[0013]In this directory type, since a related picture is arranged in one directory, the 
number of the still pictures or animations by which a list display is carried out like a list 
display type cannot become huge easily. However, it is necessary to attach a suitable 
directory name each time, and a user has forced a burden at the time of transmission. 
When the suitable directory name was not attached at the time of search, or when a user 
forgot what kind of directory name was attached, there was a problem that it became 
difficulty which should be chosen from the list displays of a directory name. 
[0014]The album type of (4) is the method of stickmg the still picture or animation 
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transmitted to the personal computer on the page created, for example with a word 
processor the same with sticking a fihn photo on pasteboard and arranging it. It can also 
be set up reproduce automatically the sound added when the added time was written in 
under the picture or the still picture was clicked when sticking on a page. 
[0015]In this album type, choice and order of the still picture or animation stuck on a 
page can be decided arbitrarily. However, when the user needed to decide choice and an 
order and a huge number of still pictures or animations were arranged, there was a 
problem that the rating which a user is forced became huge. 
[0016] 

[Problem(s) to be Solved by the Invention]As mentioned above, in the conventional 
arrangement method, it could be incompatible in arranging acquired multimedia 
information, such as picture information with notes, without applying a burden to a user, 
and searching. 

[0017]This invention is made in consideration of such a situation, and is a thing. 
The purpose is to provide the multimedia information processing unit and image 
processing device which can arrange picture information automatically by ****** and 
can be shown. 

[0018] 

[Means for Solving the Problem]In order to attain the above-mentioned purpose, this 
invention acquires acquisition time of picture information, matches this acquired time 
.-.with picture information and notes,-memorizesit, . arranges pictw^^ 
by this memorized time, and is shown. 

[0019]In this invention, it is only photoing a picture, for example, and suitable picture 

arrangement will be performed, without applying a burden to a user, since each picture is 

arranged automatically and shown according to photographing time. 

[0020]This invention acquires an acquisition position of picture information, matches this 

acquired position with picture information and notes, memorizes it, arranges picture 

information and notes by this memorized position, and is shown. 

[0021]In this invention, it is only photoing a picture, for example, and suitable picture 

arrangement will be performed, without applying a burden to a user, since each picture is 

arranged automatically and shown according to a camera station. 

[0022]This invention acquires acquisition time and an acquisition position of picture 

information, matches with picture information and notes ttiis time and position that were 

acquired, memorizes them, arranges picture information and notes by this time and 

position that were memorized, and is shown. 

[0023]In this invention, it is only photoing a picture, for example, and suitable picture 
arrangement will be performed, without applying a burden to a user, since each picture is 
arranged automatically and shown according to photographing time and a camera station. 
[0024]This invention manages schedule information of an individual or a group. 
Schedule information selected from these schedule information to manage is further 
matched with picture information and notes, and is memorized, and it is preferred for this 
schedule information to arrange picture information and notes, and to show. 
[0025]In this invention, it becomes possible to arrange a picture in a suitable part on a 
schedule of an individual or a group automatically, and to show it to it further, for 
example etc. 



Alston & Bird 



-3- 



[0026] 

[Embodiment of the Invention]Hereafter, the embodiment of this invention is described 
with reference to drawings. 

[0027] (A 1st embodiment) A 1st embodiment of this invention is described first. 
[0028] Drawing 1 is a figure showing the outline composition of the multimedia 
information processing unit concerning a 1st embodiment of this invention, and drawing 
2_is a figure showing an example of the appearance of the multimedia information 
processing unit of this 1st embodiment. 

[0029] As shown in drawmg 1, the multimedia information processing unit of this 1st 
embodiment, The image acquisition section 1 which gains picture information, such as a 
still picture or an animation, and the comment section 2 which attaches notes to the 
picture information which the image acquisition section 1 gained, the time which 
calculates the time which gained picture information ~ calculation ~ with the part 3, and 
the still picture or animation gained by the image acquisition section 1. the notes added 
by the comment section 2, and time ~ calculation ~ with the multimedia correspondence 
storage parts store 4 which makes the time calculated in the part 3 correspond, and 
memorizes it. the information memorized by the multimedia correspondence storage 
parts store 4 ~ time - calculation ~ with the arrangement presentation part 5 which the 
time calculated in the part 3 arranges and is shown. It comprises the record'presentation 
switch part 6 which chooses whether the multimedia information memorized [ whether 
muhimedia information (picture information to which notes (it is hereafter called media 
-jnformation)-of a-figure, a-sound,-a handwriting memo,- etc, -were given.)^^^^^ 
by the multimedia correspondence storage parts store 4 is shown. Hereafter, each part 
including correspondence with drawing 2 is explained. 

[0030]The image acquisition section 1 gains picture information, such as a still picture 
and an animation. For example, it comprises a camera, a TV receiver, a video recovery 
machine, etc. It is constituted from the camera la attached to the main part, and the 
shutter release lb which specifies the timing which incorporates a still picture when 
gaining a still picture by the example of drawing 2 . 

[0031]The comment section 2 inputs notes of a figure, a sound, a handwriting memo, etc. 
to the picture which the image acquisition section 1 acquired. It consists of a graphical 
input means which comprises tiie cursor 2a, button 2b, and the pointer 2c for inputting a 
figure, and a sound input means which comprises the microphone 2d for inputting a 
sound in the example of drawing 2 . By leaning the cursor 2a which constitutes a 
graphical input means vertically and horizontally, the pointer 2c is moved on the 
arrangement presentation part 5a which comprises a liquid crystal display. Button 2b 
specifies the position of the pointer 2c on a screen by clicking. The microphone 2d inputs 
the soimd of a sound or the circumference. 

[0032]time - calculation - the part 3 calculates the date and time, in order to identify 
when the picture information which the image acquisition section 1 gained was gained. 
For example, it comprises a digital clock. 

[0033]the picture information fi:om which the image acqmsition section 1 gained the 
multimedia correspondence storage parts store 4, tiie media information which the 
comment section 2 inputted to the picture information, and time — calculation - time 
which the part 3 calcxilated and when the image acquisition section 1 gained picture 
information is made a set, and is matched and memorized. For example, it comprises a 
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hard disk with a file system. 

[0034]the information the arrangement presentation part 5 was remembered to be by the 
multimedia correspondence storage parts store 4 ~ time - calculation — it arranges and 
shows using the time which the part 3 calculated. It is constituted from the liquid crystal 
display 5a for showing picture information and media information, and the loudspeaker 
5b for showing a sound by the example of drawing 2 . 

[0035]It is chosen whether record/presentation switch part 6 presents the information 
memorized [ whether multimedia information is recorded and ] by the multimedia 
correspondence storage parts store 4. It is constituted from the example of drawing 2 bv 
the slide switch 6a, multimedia information is acquired in the state where it slid to the 
left, and information is shown in the state where it slid to the right. 
[0036]Hereafter, a still picture is explained taking the case of a figure and a sound as 
media information which the comment section 2 inputs as a picture which the image 
acquisition section 1 acquires, 

[0037]If the slide switch 6a is made to slide to the left, this multimedia information 
processing unit will acquire multimedia information, and will be in the recorded state 
memorized to the multimedia correspondence storage parts store 4. 
[0038]First, a still picture is gained using the camera la. Acquisition of this still picture is 
performed when the shutter release lb is pushed. The gained still picture is memorized by 
the multimedia correspondence storage parts store 4, and the memorized still picture is 
displayed on the liquid crystal display 5a. In the multimedia correspondence storage parts 
-store4, drgMongJ^ 

explains only memory of a still picture here. As it incorporates as a monochrome image, 
for example, is shown in a2, a still picture is changed into the procession of 0 and 1 , and 
is saved at a file. The still picture saved at this file is displayed on the liquid crystal 
display 5a. 

[0039]time ~ calculation - the part 3 calculates the time which gained the still picture 
with the camera la. Time calculates time and time like "18:28 35 seconds on July 7, 
1999", for example. 

[0040]Next, notes of a figure are inputted on the picture displayed by the comment 
section 2 on the liquid crystal display 5a. Here, the example which inputs a circular figure 
is taken up and explained. Drawing 4 shows signs that a circular figure is inputted, and 
explains them hereafter according to this drawing 4 . A circular center is determined as PO 
by moving the pointer 2c which leaned the cursor 2a vertically and horizontally, and was 
displayed on the liquid crystal display 5a, and pushing button 2b with desired 
coordinates. If the cursor 2a is leaned witii button 2b pushed and the pointer 2c is moved, 
as shown in drawing 4 , PO will be set as a circular center and the round shape which 
makes the radius r the line segment which connects PO and the coordinates PI of a 
pointer will be displayed on the liquid crystal display 5a, If button 2b is detached to 
adjust the radius r and fix a round shape, looking at this roimd shape, the radius of a circle 
will be fixed and the input of a figure will be completed. The multiple input of the figure 
may be carried out on the same still picture. 

[0041] While having inputted the figure, a sound is gained from the microphone 2d as 
media information. Drawing 5 is what shows signs that a soimd is inputted inputting a 
figure, The circle bl is drawn and shown in the part which is observing the still picture 
displayed on the arrangement presentation part 5, while drawing the curcle finrther, the 
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sound b2 "the camera is attached here" is uttered toward the microphone 2d, and it inputs 
as notes. The recording start of a sound is started when the circular center PO is 
determined for example, and the end of record of a sound is ended when the radius of a 
circle is fixed. When the multiple input of the figure is carried out, the multiple input also 
of the sound is carried out. 

[0042]The multimedia correspondence storage parts store 4 matches and memorizes each 
data of a still picture, a figure, a sound, and time gained as mentioned above. Signs that a 
still picture, a figure, a sound, and time are memorized as a file to the hard disk al with a 
file system are expressed with drawing 3 . An example of the contents of a still picture, a 
figure, a sound, and time is shown by this drawing 3 . 

[0043]The multimedia correspondence storage parts store 4 will generate a unique 
(meaning) data name to a file system, if a still picture is gained. For example, the value of 
a data name coxmter is made into triple figures, it connects with the character string 
"Data", and generation of a data name is performed by **************ing a counter 
after data name generation. Matching is performed by memorizing each data of a still 
picture, a figure, a sound, and time as a file which has the same data name and fi:om 
which an extension differs. There are three data names, DataOOl, Data002, and Data004, 
and the case where three still pictures are gained is expressed with the example of 
drawing 3 . 

[0044] Among these, the data with data name Data002 has time file Data002.dat with still 
picture file Data002.pic, graphic file Data002.shp, and sound file Data002.snd. The 

--€Xtension--piG"4s-m extension-vwth-which "shp"^ 
a figure, and a sound and "dat" express time. 

[0045]Explanation of the contents of each data file will memorize still picture file 
Data002.pic as data expressed with the procession of 0 and 1 as mentioned above as 
shown in a2. Graphic file Data002.shp is memorized as a numerical sequence of carrying 
out for constructing the circular center coordinates PO (138,438) and the radius r (37) 
(138, 438, 37), as shown in a3. When there are two or more figures, as shown in a2, only 
the number of a figure adds a numerical sequence back. Sound file Data002.snd 
enumerates and memorizes the integral value of -254 to 256 which carried out the A/D 
conversion of the sound inputted fi-om the microphone 2d, as shown in a4. When the 
multiple input of the figure is carried out, after adding value-255 which does not appear, 
for example as data as a punctuation mark, the value which carried out the A/D 
conversion of the sound is added. Time file Data002.dat is saved as a character string 
showing the date and time like "18:28 35 seconds on July 7, 1999", as shown, for 
example in a5. Since time is only 1 time when a still picture is gamed, only one exists 
about one data name. 

[0046]On the other hand, if the slide switch 6a is made to slide to the right, this 
multimedia information processing unit will be in the presentation state which presents 
the information memorized to the multimedia correspondence storage parts store 4. 
[0047]If it will be in a presentation state, the arrangement presentation part 5 will arrange 
the information memorized by the multimedia correspondence storage parts store 4 using 
time, and will present it on the liquid crystal display 5a. Drawing 6 shows this example 
and shows signs that the thumbnail of the still picture which the image acquisition section 
1 gained is displayed on the calendar for one month, the colunm as which a thumbnail is 
displayed ~ time — calculation -- it is displayed on the same column as the date 
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calculated by the part 3. When two or more still pictures are gained on the same day, the 
gained tune is displayed on early order from a top. For example, since it corresponds to 
data name Data002 of drawing 3 and Data002 was gained on July 7, the thumbnail image 
cl of drawing 6 is displayed on the column on July 7 of the calendar table of drawing 6 . 
[0048]In generation of a list display, about the days from 1 of the moon which was in the 
presentation state first to the end of the month, all the time memorized by the time file is 
read, it arranges in the order memorized early, and a table is made. A table is memorized 
by the multimedia correspondence storage parts store 4 by a file name called Table.tal. 
Drawing 7 shows the situation of rearrangement and shows signs that it has ranked with 
the early order of the time when three time file names were memorized sequentially from 
the top. The time file which memorized the same date is put in order by the early order of 
time from a top. 

[0049]Next, a time file name is read in a head, a still picture file with the same data name 
is searched from a multimedia correspondence storage parts store, a thumbnail image is 
generated, and it displays on the column of the same date of a calendar. It reduces to the 
generate time of a thumbnail image together, and it not only reduces a still picture, but 
displays the figvire matched with the still picture on it. Search of the figure matched with 
the still picture searches a graphic file with the same data name as a still picture file, 
restores the memorized graphic information, and draws on a still picture. For example, 
the graphic file matched with Data002.pic is Data002.shp. When it already displays on 
the same date, as shown in C2 of drawing 6. it displays in addition under the thumbnail 

-4mage-displayed-by 4hen.- - - - - - - 

[0050]Thus, it is easily discriminable by looking through the thumbnail image of a still 
picture what kind of data the user gained when. As for it, since a list display is 
automatically generated based on the time which gained the still picture, a burden is not 
placed on a user at all. 

[005 l]In a presentation state, the cursor 2a, button 2b, and the pointer 2c are used in 
order to specify the thumbnail displayed on the liquid crystal display 5a. A click of the 
thumbnail image displayed on the column on July 7 when explained taking the case of 
drawing 6 will display the picture shown in drawing 8 . In this display, not only a still 
picture but a figure is displayed. The search method of the graphic file matched with the 
still picture file is the same as the time of generating a reduction image. 
[0052]A click of the inside of the circular figure el displayed on the still picture 
displayed on drawing 8 w ill reproduce the sound e2 matched with the still picture. If the 
search method of the sound matched with the still picture is explained taking the case of 
Data002.pic, it will have the data name Data002 [ same ], and when an extension 
searches the file of snd, i.e., Data002.snd, it will carry out. When those with two or more, 
for example, the 2nd figure, are chosen, a figure searches a Data002.snd file from a head, 
and reproduces the 2nd sound field divided by pimctuation mark-255. 
[0053]Thus, by seeing the figure drawn on the still picture, by recognizing where [ in a 
still picture ] it observed, or hearing the sound ftirther related with the figure, the user can 
get more detailed notes, and he can acquire rich information rather than merely seeing a 
still picture only. 

[0Q54] Drawing 9 is the flow chart which summarized the above operation. According to 
this flow chart, the flow of processing is explained below. 

[0055]First, from the state of the slide switch 6a, it investigates whether this multimedia 
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information processing unit is a recorded state, in the case of a recorded state, progresses 
to Step A2, and, in the case of the presentation state which is not so, progresses to Step 
A6 (StepAl). 

[0056]If it stands by that the shutter release lb is pushed and the shutter release lb is 
pushed in order to gain a still picture, in the case of a recorded state, a still picture will be 
incorporated from the camera la, it will memorize it to the multimedia correspondence 
storage parts store 4, and will be shown to the liquid crystal display 5 a (Step A2). 
[0057]Next, it investigates whether the shutter release 7 was pushed by step A3, when not 
pushed, it returns to Step A2 and acquisition of a preview image is continued, and when 
pushed, the still picture which he followed to step A4 and was gained at the end is 
memorized to the multimedia correspondence storage parts store 4 as it is. this time ~ 
time -- calculation - the part 3 calculates the time when this still picture was gained (step 
A4). 

[0058]Then, will be in the state where notes can be inputted on the gained still picture, 
and it will be judged whether the shutter release lb was pushed by step A5, When 
pushed, the input of notes is suspended and it returns to Step Al, and when not pushed, it 
progresses to Step A13 and media information, such as a figure and a sound, is inputted 
as notes. 

[0059]And when there is an input of media information, such as this figure, sound, etc., 
in order to memorize these to the multimedia correspondence storage parts store 4 and 
input notes continuously, it returns to step A5. 
. _[.0.0.6fl] When itxhangesla a-presentatiom state^^M swLtch-da by a memory state, 

immediately, a state does not change, but after the shutter release lb is clicked by step 
A5, it changes to a presentation state at Step A. 

[0061]on the other hand ~ Step Al ~ the case of a presentation state - time ~ calculation 
- a table is created in the early order of the time calculated in the part 3 (Step A6), and a 
calendar list is displayed on the liquid crystal display 5a. The thumbnail image of the still 
picture memorized by this calends: list at the multimedia correspondence storage parts 
store 4 is displayed on the column of the date which gained the still picture (Step A7). 
[0062]Then, it will be in the state where a thumbnail image can be clicked, and when 
clicked, the enlarged display of the picture clicked by progressing to step A9 is carried 
out, when that is not right, it returns to Step Al and a calendar display is continued (Step 
AS). 

[0063]It investigates whether it was clicked in the enlarged display by step A9, when 
clicked, it progresses to Step All, and when that is not right, it returns to step A9 and an 
enlarged display is continued (Step A 10). 

[0064] When it progresses to Step Al 1, it is investigated whether the click took place on 
the figure, When it is not figure up, it returns to Step Al and returns to a calendar list 
display, and after it reproduces the sound matched with the figure (Step A 12) and 
reproduction finishes, in on a figure, it returns to step A9, and it continues an enlarged 
display. 

[0065]Thus, accordmg to the multimedia information processing unit of this 1st 
embodiment. Effective use of the data stored every day will be achieved without forcing 
a user a complicated view creation procedure by creating the view which arranges 
automatically multimedia information with the various structures which are not only still 
pictures by time and with which it not only can acquire it easily, but it searches it. 
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[0066]The modification which can be given to the multimedia infonnation processing 
unit of this 1st embodiment is shown below. 

[0067]First, although the picture infonnation gained by this 1st embodiment was a still 
picture, it may not be limited to this but an animation may be sufficient as it. 
[0068]In order to make it intelligible what kind of image to be captured although nothing 
was displayed on the screen when gaining a still picture by this 1st embodiment, on the 
liquid crystal display 5a, a preview image may be displayed or a finder for exclusive use 
may be provided. 

[0069] Although the media information which the comment section 2 inputs by this 1st 
embodiment was a circular figure and a sound, if media information is not limited to this 
and the particular part of a still picture is explained, it is [ anything ] good. For example, 
the form of a figure may use not only a round shape but a rectangle, an arrow, etc. As an 
example of media information other than a figure, a text and a handwriting memo may be 
used. In the case of a text, a keyboard is attached to a device, for example, and after 
specifying with a pointer the position on the still picture which inputs a text, it should just 
be inputted. What is necessary is to stick a transparent pressure-sensitive type tablet, for 
example on the liquid crystal display 5a, to trace an it top with a pen, and just to input, 
when inputting a handwriting memo. 

[0070] Although the sound inputted by this 1st embodiment was matched together with 
the circular figure and inputted, the media information inputted together is not restricted 
to a circular figure. A sound may input a figure etc. independently rather than is 

. necessanly_required.-A.sQundmay-b&-matched-and.inputtedJa.the whole still p^ 

[0071]The media information inputted by this 1st embodiment may be a link. For 
example, a figure and other still pictures may be associated so that it may display that 
other still pictures click a figure. When a figure is inputted like a 1st embodiment, the 
input was completed, acquisition of a still picture is started immediately and acquisition 
of a still picture is completed after specifying the input of a link with a button etc., the 
input of a link, A figure and a still picture are associated using the file name memorized 
by the multimedia correspondence storage parts store. Correlation is performed to a 
graphic file by writing in the file name of the still picture of a link destination. 
[0072] Although one thumbnail image was equivalent to one still picture in the calendar 
list display of this 1st embodiment. The still picture memorized by the time it was from 
the recorded state in the presentation state may be packed as one unit (session), and the 
still picture gained first, for example may be used as a picture representing a session as a 
thumbnail image. In this case, when carrying out an enlarged display, the still picture and 
media information corresponding to the clicked still picture are (Usplayed, page turning 
over may be carried out with the cursor 2a, and the still picture and media information 
after the 2nd sheet may be displayed. 

[0073] Although the still picture gained first was used as a representation screen in the 
thumbnail indication usmg this session, it may not restrict to this and a thumbnail image 
may be displayed every five sheets, for example. In this case, when the thumbnail image 
of the 5th sheet is specified by a presentation state, the enlarged display of the still picture 
of the 5th sheet may be carried out automatically. 

[0074]Although the thimibnail image of notes inputted as the still picture on it was used 
for the calendar list display displayed by this 1st embodiment, if it turns out that there is 
information, it not only this but is [ anytiiing ] good. For example, the icon which shows 
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that there is a still picture may be displayed, or a file name may be displayed. 
[0075]If the calendar list display displayed by this 1st embodiment is the method of 
arranging and showing the picture information and media information which were 
memorized by the multimedia correspondence storage parts store 4 for one month 
although it was a unit using time, it is [ anything ] good. For example, it may display in 
order of time per a unit or one week like schedule software for one day. Or only a still 
picture with the same title may be looked through in order of the recorded time. Thereby, 
it can respond to a customer list etc. When a text is inputted as media information, the list 
display only of the still picture containing the character string specified by a user may be 
carried out to the recorded order of time. 

[0076] Although the arrangement presentation part 5 of this 1st embodiment displayed the 
calendar list on the liquid crystal display 5a with which the device main frame was 
equipped. The file memorized to the multimedia correspondence storage parts store 4 
may be transmitted, for example to a personal computer, may be changed into HTML 
form, and a list display may be carried out on a personal computer. A personal computer 
possesses the browser which presents the data changed into HTML form. Thus, since the 
HTML browser which has spread widely by changing into HTML can be used, the user 
can see a calendar list display, without newly installing a browser for exclusive use. 
[0077] Although the enlarged display of this 1st embodiment displayed a still picture and 
1 page of media information at a time on the liquid crystal display 5a, a title etc. may be 
displayed simultaneously. Drawing 10 shows an example of an enlarged display and the 
.-title, .the_fojit,...the4>icture^f.a-backgromd,. and recording .date_are .specified. In this 
example, the text is also inputted as media information and it is displayed beside the still 
picture. Selection of the picture of a title, a font, and a background is asked to a user 
about the still picture gained newly, for example, when it changes fix>m a recorded state 
to a presentation state. Thus, the list nature of the arranged data increases further by 
inputting a title etc. 

[0078] When displaying a title by the enlarged display of this 1st embodiment, the layout 
of a still picture, a text, etc. was immobilization, but a layout may be changed depending 
on a title etc. For example, there is a tendency to input the thing about in the hall [ which 
records a display object etc. in an exhibition or an exposition ], and the title with a short 
text which mainly gains many still pictures. Therefore, in order to arrange these legible 
and to show them when choosing a title, and the category of "an exhibition and an 
exposition" is chosen, as shown in drawing 11 . the layout which has arranged the text 
under a still picture is chosen automatically, and is used. Or when recording a meeting, 
tiiere is a tendency to input a lot of texts supplementary to the still picture to gain. 
Therefore, in order to arrange these legible and to show them when choosing a titie, and 
the category a "meeting" is chosen, as shown in drawing 12, the layout which matched 
the still picture and the text is chosen automatically, and is used. In addition, there may 
be categories, such as a "travel diary" and a "guest person." It may change so that the 
picture of a background may become suitable by the contents. The picture of a suitable 
background may be automatically chosen as the season according to the season which 
recorded the still picture. One spot's picture may be chosen at random and may be 
arranged. Thereby, an enlarged display screen with originality can be generated 
automatically. 

[0079]Matching of a category and a layout prepares the table which matched the category 



Alston & Bird 



-10- 



and the layout by 1 to 1, for example, and when a category is specified, it should just 
generate an enlarged display using the layout corresponding to it. Thereby, the kind of 
data to record enables it to generate the optimal enlarged display automatically. 
[0080] Although it clicked the circular inside of a figure, if selection of the figure in the 
enlarged display of this 1st embodiment is the method of choosing a figure, it not only 
this but is [ anything ] good. For example, the line segment and circle top which 
constitute a figure may be clicked and chosen. 

[0081](A 2nd embodiment), next a 2nd embodiment of this invention are described. 
[0Q82] Drawing 13 is a figure showing the outline composition of the multimedia 
information processing unit concerning a 2nd embodiment of this invention, and drawing 
14 is a figure showing an example of the appearance of the multimedia information 
processing unit of this 2nd embodiment. 

[0083]Using GPS (GlobalPositioning System) etc., the multimedia information 
processing unit of this 2nd embodiment acquires the position which gained picture 
information, and a position arranges the acquired multimedia information and it presents 
it. 

[0084]That is, although arrangement presentation of the acquired multimedia information 
was carried out in a 1st above-mentioned embodiment by the time which gained picture 
information, not time but a position is used for this 2nd embodiment as information used 
in order to carry out arrangement presentation. 

[0085] Drawing 13 g ives identical codes to drawing 1 and an identical configuration 
.denient,.and.gixesjiumeialsjonly.lQ-ajdifEeEentcjompo^^ the time 

of drawing 1 - calculation - the position which there is no part 3 and comprises GPS etc. 
— calculation — the composition which added the part 7 is shown, a position - calculation 
~ a portion which is different here smce the composition of those otiher than part 7 is the 
1st embodiment and identical configuration, i.e., a position, - calculation — only the part 
7 is explained including correspondence with drawing 14 . 
[0086]a position - calculation ~ the part 7 calculates a position, in order to identify 
whether tiie picture information which the image acquisition section 1 gained was gained 
at which place. It is constituted from the example of dravying 14b v GPS7a. 
[0087]Hereafl:er, a still picture is explained like a 1st embodiment taking the case of a 
figure and a sound as media information which the comment section 2 inputs as a picture 
which the image acquisition section 1 acquires. 

[0088]First, a still picture is gained like a 1st embodiment using the camera la. The 
gained still picture is memorized by the multimedia correspondence storage parts store 4, 
and the memorized still picture is displayed on the liquid crystal display 5a. 
[0089]a position - calciUation ~ the part 7 calculates the position from which the image 
acquisition section 1 gained the still picture. A position is calculated like north latitude 
35-degree 32-minute and 45 seconds, and east longitude 139-degree "41 minutes and 42 
seconds", using the longitude and latitude which were obtained by GPS. 
[0090]The multimedia correspondence storage parts store 4 matches and memorizes each 
data of a still picture, a figure, a sound, and a position. In the multimedia correspondence 
storage parts store 4, drawing 15 shows signs that a still picture, a figvire, and a position 
are memorized, gives identical codes to a 1st embodiment and identical parts, and gives 
new numerals to a different portion. Each data of a still picture, a figure, a sound, and a 
position is memorized as a file by the hard disk al which has a file system like a 1st 
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embodiment. 

[0091]The file name of a position file connects and generates "pos" which is an extension 
showing a position to data names, such as "Data002", as well as other information. It 
matches with other information by attaching the same data name as other information to 
the file name of a position file. The contents of position file Data002.pos are stored by the 
character string of north latitude 35-degree 32-minute and 45 seconds, and east longitude 
139-degree "41 minutes and 42 seconds", as shown in fl. Operation of others of a 
memory state is the same as that of a 1st embodiment. 

[0092]Next, only a different portion fi-om a 1st embodiment is explained about operation 
of a 2nd embodiment at the time of an instruction state. 

[0093]If it will be in a presentation state, the arrangement presentation part 5 will arrange 
the information memorized by the multimedia correspondence storage parts store 4 using 
a position, and will present it on the liquid crystal display 5a. Drawing 16 shows this 
example and shows signs that a thumbnail image with the media information which the 
still picture which the image acquisition section 1 gained, and the comment section 2 
inputted is displayed on a map. 

[0094]The map file "map.pic" in which map information was stored beforehand is 
memorized by the multimedia correspondence storage parts store 4, and the display of a 
map displays this. A position file "map.pos" for the position information which shows the 
range of the position of a map to match is shown in a map file. The position of the point 
P2 in drawing 16 a nd the point P3, for example, north latitude 35-degree 33-minute and 
-.05-seconds, and east longitude 1 39=degree "41 minutes and 09 seconds", and. north 
latitude 35-degree 32-minute and 33 seconds, and east longitude 139-degree "42 minutes 
and 05 seconds" are memorized. 

[0095]Where [ on the liquid crystal display 5a ] it displays asks for a thumbnail image as 
follows, drawing 17 - P2 (X2, Y2), P3 (X3, Y3), and a position position ~ calculation - 
the position P4 (X4, Y4) calculated in the part 7 shows the position displayed on the 
liquid crystal display 5a. If the coordinate value on the liquid crystal display 5a of these 
P2, P3, and P4 is set to p2 (x2, y2), p3 (x3, y3), and p4 (x4, y4), the coordinate value of 
P4 will be calculated by the following (1) - (2) formulas. 
[0096] 

x4=(x3-x2) *(X3-X4)/(X3-X2) - (1) Formula y4=(y3-y2) *(Y3-Y4)/(Y3-Y2) - If (2) 
type thumbnail image is clicked by the same method as a 1st embodiment, The enlarged 
display of the media information drawn the still picture corresponding to a thumbnail 
image and on it is carried out. Operation of others of a presentation state is the same as 
that of a 1st embodiment. 

[00971 Drawing 18 is tiie flow chart which summarized the above operation, and explains 
the flow of processing according to this flow chart below. This drawing 18 g ives identical 
codes to drawing 9 and identical parts which are the flow charts of a 1st embodiment, 
gives numerals only to a different portion newly, and explains only a different portion 
fi-om a 1st embodiment here. 

[0098]if the shutter release lb is pushed by step A3 and a still picture is incorporated ~ a 
position ~ calculation - the position fi-om which the still picture was gained is calculated 
in the part 7 using GPS etc. (Step Bl). The remaining processing operation at the time of 
a memory state is the same as that of a 1st embodiment. 

[0099] When it progresses to step B-2 by a presentation state, a thumbnail image with the 
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media information inputted by the still picture gained by the image acquisition section 1 
on the map displayed on the arrangement presentation part 5 and the comment section is 
displayed on the coordinates corresponding to the position from which the still picture 
was gained. The remaining processing operation at the time of a presentation state is the 
same as that of a 1st embodiment. 

[0100]Thus, according to the multimedia information processing unit of this 2nd 
embodiment. Effective use of the data stored every day will be achieved without forcing 
a user a complicated view creation procedure by creating the view which arranges 
automatically multimedia information with the various structures which are not only still 
pictures by a position and with which it not only can acquire it easily, but it searches it. 
[OlOljThe modification which can be given to the muhimedia information processing 
unit of this 2nd embodiment is shown below. 

[0102]first, the position of this 2nd embodiment ~ calculation ~ although the part 7 
calculated the latitude and longitude on the earth using GPS, if a position understands it, 
it is not limited to this but is [ anything ] good. For example, it may calculate having 
gained the still picture within a fixed distance from the base station using a radio wave 
beacon. In addition, an infrared beacon, a geomagnetism sensor, an altimeter, etc. may be 
used, or a user may read the bar code stretched by the wall and may calculate a position. 
It is good also considering this as a position to carry out character recognition of the 
picture projected on the preview screen, and gain a room number. 
[0103]moreover - although the picture which the arrangement presentation part 5 
-presents-by4his.2nd-embodimentJwasa-map-r-r..a.positioa--=£alc. 
good if the position calculated in the part 7 can be used. For example, a drawing, a train 
line figure, etc. of a factory may be sufficient. Or a list display may be carried out 
according to conference room. For example, the list display of all the data of the past 
gained in A conference room may be put in order and carried out to the early order of 
time. 

[0104]moreover ~ this 2nd embodiment - a position - calculation - although the part 7 
calculated the position and the position arranged and showed the arrangement 
presentation part 5 - a 1st embodiment - like - time ~ calculation ~ a part may be 
provided, time may be calculated, and the arrangement presentation part 5 may also 
arrange and present time. In this case, the switch which specifies whether time shows 
whether a position shows for example, is formed. 

[0105](A 3rd embodiment), next a 3rd embodiment of this invention are described. 
[0106] Drawing 19 is a figure showing the outline composition of the multimedia 
information processing unit concerning a 3rd embodiment of this invention, and drawing 
20 is a figure showing an example of the multimedia information processing unit of this 
3rd embodiment. 

[0107]The information inputted into the schedule software of the individual or the group 
arranges the multimedia information processing unit of this 3rd embodiment, and it is 
shown. 

[0108]Namely, although arrangement presentation of the acquned multimedia 
information was carried out in a 1st above-mentioned embodiment and a 2nd 
embodiment by the time and the position which gained picture information. The schedule 
of an individual or a group is fijrther used for this 3rd embodiment as information used in 
order to carry out arrangement presentation. 



Alston & Bird 



-13- 



[0109] Drawing 19 g ives identical codes to drawing 1 and an identical configuration 
element, and gives numerals only to a different component newly. Drawing 19 shows the 
composition which added the schedule management section 8 and the schedule selecting 
part 9 to the component of drawing 1. and accepts and explains the portion 8 different 
here from a 1st embodiment, i.e., a schedule management section, and the schedule 
selecting part 9. 

[01 10]The schedule management section 8 comprises a hard disk and a file system, for 
example, manages the schedule element of "holding a conference from <1>:00 to <2>:00 
on August 2" as one file. 

[01 1 l]the time out of the schedule element in which the schedule selecting part 9 is 
managed by the schedule management section 8 ~ calculation - the element in which the 
time calculated in the part 3 is contained is chosen. 

[01 12]Hereafter, a still picture is explained like a 1st embodiment taking the case of a 
figure and a sound as media information which the comment section 2 inputs as a picture 
which the image acquisition section 1 acquires. 

[01 13]First, a still picture is gained like a 1st embodiment using the camera la. the 
gained still picture is memorized by the multimedia correspondence storage parts store 4, 
and the memorized still picture is displayed on the liquid crystal display 5a -- time - 
calculation - the part 3 calculates the time when the still picture was gained. 
[01 14]As for the schedule management section 8, as shown in drawing 21 , the schedule 
of an individual or a group manages one schedule element as one file. A schedule 
.elementis a-gr.aup_of thejdate,-time,..and.a..schedule..name,.and " 1 Q:QO on My 7 to 1 2:00 
and a meeting" are memorized in the example of drawing 21 b v the file "Schedule.OOl ." 
When a new schedule element is generated, the extension of a file name is 
**************edanda schedule element is generated so that a file name may not 
overlap, 

[01 15] When the timing which a schedule generates makes the slide switch of 
record/directions switch part 6 3 ream type and "Schedule" is chosen for example, it is 
made to change to the schedule generation screen shown m drawing 22 . In the case of 
this example, it is in the beginning which changed at the place whose pointer is time, and 
the cursor 2a is made to go up and down, the date is changed, and the date on "July 7, 
1999" is set up. If the cursor 2a is moved to the right, a pointer will move to the next 
setting out, schedule start time can be set up now, and "10:00 to 12:00" is set up. From 
the lists prepared beforehand, a schedule name makes the cursor 2a go up and down, and 
is chosen. After all the setting out finishes, a schedule is made to become final and 
conclusive by pushing the button 2a, a "meeting" is set up, and the schedule management 
section 8 generates a file newly, and memorizes these contents. 
[01 16]the time out of the schedule element in which the schedule selecting part 9 is 
managed by the schedule management section 8 ~ calculation - the thing containing the 
time which the part 3 calculated is chosen, the example of drawing 22 - time ~ 
calculation, when the part 3 measures "10:42 56 seconds on July 7, 1999", Since this time 
is contained in "the schedule at "10:00 to [ July 7", 1999 ] 12:00 on July 7, 1999 
memorized by "Schedule.OOl"", this schedule element is chosen. 
[01 17]Like a 1st embodiment, the multimedia correspondence storage parts store 4 also 
matches and memorizes a schedule element while memorizing a still picture, a figure, a 
sound, and time. Matching of a schedule element generates the file in which an extension 
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has ".scd" with the same data name like a 1st embodiment, and memorizes the file name 
memorized by the schedule management section 8 to this file. When a still picture file 
name is "Data002.pic", the file name of a schedule element is set to "Data002.scd", and 
"Schedule.OOl" is written in the file and it is memorized. Operation of others of a 
memory state is the same as that of a 1st embodiment. 

[01 1 8]Next, only a different portion fi-om a 1st embodiment is explained about operation 
of a 3rd embodiment at the time of a presentation state. 

[01 19]In a presentation state, if a calendar display as shown in drawing 23 is performed 
like drawing 6 and a thumbnail image is clicked, the enlarged display shown, for example 
in drawing 8 w ill be carried out. When creating an enlarged display and there is a 
schedule containing the time which gained the still picture, the title of a schedule, other 
time, a place, etc. are displayed on the title of an enlarged display. 
[0120]The judgment of the existence of the schedule containing the time which gained 
the still picture. In this example, the display of data name Data002, After reading a file 
name called Schedule.OOl first written in into file DataOOl.scd, The schedule of file 
Schedule.OOl memorized by the schedule management section 9 is read, and it detects 
that Data002 is matched with the schedule at 12:00 from 10:00 on July 7, 1999. 
Operation of others of a presentation state is the same as that of a 1st embodiment. 
[0121] Drawing24 is the flow chart which summarized the above operation, and explains 
the flow of processing according to this flow chart below. This drawing 24 g ives identical 
codes to drawing 9 a nd identical parts which are the flow charts of a 1st embodiment, 

_givies numerals only tQ.a -different-portion newly, and explains only.a-different portion 

firom a 1st embodiment here. 

[0122]First, the state of the slide switch 6a is investigated, and it detects whether it is in 
which state among three, record, presentation, and a schedule input, in the case of a 
recorded state, progresses to Step A2, in the case of a presentation state, progresses to 
step A3, and, in the case of a schedule input state, progresses to Step C4 (Step CI). 
[0123]In the case of a schedule input state, a schedule is inputted and it returns to Step CI 
(Step C4). 

[01 24] After in the case of a recorded state incorporating a still picture, inputting notes 
like a 1st embodiment, pushing the shutter release lb by step A5 and completing the 
input of notes, The schedule containing the time when the schedule selecting part 9 
recorded the still picture is chosen firom the schedule management section 8, and it 
memorizes to the multimedia correspondence storage parts store 4, and returns to Step CI 
(Step A2). In the case of a presentation state, the same operation as a 1st embodiment is 
carried out. 

[0125]Thus, according to the multimedia information processing unit of tiiis 3rd 
embodiment. The data stored every day can be searched efficiently, without forcing a 
user a complicated view creation procedure by creating automatically the view which it 
not only can acquire easily multimedia information with the various structures which are 
not only still pictures, but arranged it in schedule form. 

[0126]The modification which can be given to the multimedia information processing 
imit of this 3rd embodiment is shown below. 

[0127]First, although arrangement presentation was carried out in this 3rd embodiment in 
connection with the time which gained the still picture, and the time of a schedule, it is [ 
anything ] good if based on the relation of the annotation information attached not only to 
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this but to the still picture, and the element of a schedule. For example, when the position 
which recorded the still picture when the position which recorded the still picture was 
acquired like a 2nd embodiment and a place was described by the schedule, and the 
position of a schedule are the same, it may be made to display a thumbnail image. 
[0128] 

[Effect of the Invention]If the multraiedia information processing unit of this invention is 
used as explained in full detail above, In order not to force a user a complicated view 
creation procedure by creating the view which it not only can acquire multimedia 
information with structure easily, but time, a position, and a schedule arrange 
automatically and it searches. Without being conscious of how data is stored, data can be 
stored effectively and the user can search it. 
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